On probabilistic inference approaches to stochastic optimal control

While stochastic optimal control, together with associate formulations like Reinforcement Learning, provides a formal approach to, amongst other, motor control, it remains computationally challenging for most practical problems. This thesis is concerned with the study of relations between stochastic optimal control and probabilistic inference. Such dualities { exempli ed by the classical Kalman Duality between the Linear-Quadratic-Gaussian control problem and the filtering problem in Linear-Gaussian dynamical systems { make it possible to exploit advances made within the separate fields. In this context, the emphasis in this work lies with utilisation of approximate inference methods for the control problem. Rather then concentrating on special cases which yield analytical inference problems, we propose a novel interpretation of stochastic optimal control in the general case in terms of minimisation of certain Kullback-Leibler divergences. Although these minimisations remain analytically intractable, we show that natural relaxations of the exact dual lead to new practical approaches. We introduce two particular general iterative methods ψ-Learning, which has global convergence guarantees and provides a unifying perspective on several previously proposed algorithms, and Posterior Policy Iteration, which allows direct application of inference methods. From these, practical algorithms for Reinforcement Learning, based on a Monte Carlo approximation to ψ-Learning, and model based stochastic optimal control, using a variational approximation of posterior policy iteration, are derived. In order to overcome the inherent limitations of parametric variational approximations, we furthermore introduce a new approach for none parametric approximate stochastic optimal control based on a reproducing kernel Hilbert space embedding of the control problem. Finally, we address the general problem of temporal optimisation, i.e., joint optimisation of controls and temporal aspects, e.g., duration, of the task. Specifically, we introduce a formulation of temporal optimisation based on a generalised form of the finite horizon problem. Importantly, we show that the generalised problem has a dual finite horizon problem of the standard form, thus bringing temporal optimisation within the reach of most commonly used algorithms. Throughout, problems from the area of motor control of robotic systems are used to evaluate the proposed methods and demonstrate their practical utility

Evidence for sex-specific genetic architectures across a spectrum of human complex traits

BACKGROUND: Sex differences are a common feature of human traits; however, the role sex determination plays in human genetic variation remains unclear. The presence of gene-by-sex (GxS) interactions implies that trait genetic architecture differs between men and women. Here, we show that GxS interactions and genetic heterogeneity among sexes are small but common features of a range of high-level complex traits. RESULTS: We analyzed 19 complex traits measured in 54,040 unrelated men and 59,820 unrelated women from the UK Biobank cohort to estimate autosomal genetic correlations and heritability differences between men and women. For 13 of the 19 traits examined, there is evidence that the trait measured is genetically different between males and females. We find that estimates of genetic correlations, based on ~114,000 unrelated individuals and ~19,000 related individuals from the same cohort, are largely consistent. Genetic predictors using a sex-specific model that incorporated GxS interactions led to a relative improvement of up to 4 % (mean 1.4 % across all relevant phenotypes) over those provided by a sex-agnostic model. This further supports the hypothesis of the presence of sexual genetic heterogeneity across high-level phenotypes. CONCLUSIONS: The sex-specific environment seems to play a role in changing genotype expression across a range of human complex traits. Further studies of GxS interactions for high-level human traits may shed light on the molecular mechanisms that lead to biological differences between men and women. However, this may be a challenging endeavour due to the likely small effects of the interactions at individual loci. ELECTRONIC SUPPLEMENTARY MATERIAL: The online version of this article (doi:10.1186/s13059-016-1025-x) contains supplementary material, which is available to authorized users

Imputation of DNA Methylation Levels in the Brain Implicates a Risk Factor for Parkinson's Disease

Understanding how genetic variation affects intermediate phenotypes, like DNA methylation or gene expression, and how these in turn vary with complex human disease provides valuable insight into disease etiology. However, intermediate phenotypes are typically tissue and developmental stage specific, making relevant phenotypes difficult to assay. Assembling large case–control cohorts, necessary to achieve sufficient statistical power to assess associations between complex traits and relevant intermediate phenotypes, has therefore remained challenging. Imputation of such intermediate phenotypes represents a practical alternative in this context. We used a mixed linear model to impute DNA methylation (DNAm) levels of four brain tissues at up to 1826 methylome-wide sites in 6259 patients with Parkinson’s disease and 9452 controls from across five genome-wide association studies (GWAS). Six sites, in two regions, were found to associate with Parkinson’s disease for at least one tissue. While a majority of identified sites were within an established risk region for Parkinson’s disease, suggesting a role of DNAm in mediating previously observed genetic effects at this locus, we also identify an association with four CpG sites in chromosome 16p11.2. Direct measures of DNAm in the substantia nigra of 39 cases and 13 control samples were used to independently replicate these four associations. Only the association at cg10917602 replicated with a concordant direction of effect (P = 0.02). cg10917602 is 87 kb away from the closest reported GWAS hit. The employed imputation methodology implies that variation of DNAm levels at cg10917602 is predictive for Parkinson’s disease risk, suggesting a possible causal role for methylation at this locus. More generally this study demonstrates the feasibility of identifying predictive epigenetic markers of disease risk from readily available data sets

An Approximate Inference Approach to Temporal Optimization in Optimal Control

Algorithms based on iterative local approximations present a practical approach to optimal control in robotic systems. However, they generally require the temporal parameters (for e.g. the movement duration or the time point of reaching an intermediate goal) to be specified a priori. Here, we present a methodology that is capable of jointly optimizing the temporal parameters in addition to the control command profiles. The presented approach is based on a Bayesian canonical time formulation of the optimal control problem, with the temporal mapping from canonical to real time parametrised by an additional control variable. An approximate EM algorithm is derived that efficiently optimizes both the movement duration and control commands offering, for the first time, a practical approach to tackling generic via point problems in a systematic way under the optimal control framework. The proposed approach, which is applicable to plants with non-linear dynamics as well as arbitrary state dependent and quadratic control costs, is evaluated on realistic simulations of a redundant robotic plant

Genetic determination of height mediated mate choice

BACKGROUND: Numerous studies have reported positive correlations among couples for height. This suggests that humans find individuals of similar height attractive. However, the answer to whether the choice of a mate with a similar phenotype is genetically or environmentally determined has been elusive. RESULTS: Here we provide an estimate of the genetic contribution to height choice in mates in 13,068 genotyped couples. Using a mixed linear model we show that 4.1 % of the variation in the mate height choice is determined by a person’s own genotype, as expected in a model where one’s height determines the choice of mate height. Furthermore, the genotype of an individual predicts their partners’ height in an independent dataset of 15,437 individuals with 13 % accuracy, which is 64 % of the theoretical maximum achievable with a heritability of 0.041. Theoretical predictions suggest that approximately 5 % of the heritability of height is due to the positive covariance between allelic effects at different loci, which is caused by assortative mating. Hence, the coupling of alleles with similar effects could substantially contribute to the missing heritability of height. CONCLUSIONS: These estimates provide new insight into the mechanisms that govern mate choice in humans and warrant the search for the genetic causes of choice of mate height. They have important methodological implications and contribute to the missing heritability debate. ELECTRONIC SUPPLEMENTARY MATERIAL: The online version of this article (doi:10.1186/s13059-015-0833-8) contains supplementary material, which is available to authorized users